Dan's Tech Blog

Beyond mere storage efficiency, the spatial transformation enables unprecedented democratization of knowledge access: whereas a gymnasium-sized library requires dedicated building infrastructure, climate control, cataloging systems, and professional staff, the equivalent knowledge density fits on a consumer-grade GPU or even high-end laptop storage. This shift eliminates the traditional tradeoffs between knowledge breadth and physical accessibility, allowing individuals to query the equivalent of a substantial research library from devices that fit in a backpack, fundamentally altering the economics and logistics of information distribution compared to the physical constraints that have governed libraries for millennia.

13GB English Wikipedia vs 4GB Llama2 Model

The entire English Wikipedia, comprising billions of words across millions of articles, occupies approximately 13 GB when compressed, yet a quantized 7B parameter LLaMA model requires only around 4 GB of storage after 4-bit quantization—less than one-third the size. This remarkable compression differential illustrates a fundamental shift in knowledge representation: while Wikipedia stores raw text that must be searched and read linearly, the LLaMA model encodes linguistic patterns, relationships, and semantic understanding within its parameter weights, enabling it to generate contextually relevant responses rather than simply retrieve stored text. Despite its smaller footprint, the 13B parameter LLaMA model demonstrated performance comparable to GPT-3 (175B parameters) across most NLP benchmarks, showing that efficient parameter encoding can rival models more than ten times larger. The 4-bit quantization process reduces the original 13 GB memory requirement of an uncompressed 7B model to just 4 GB, making these capabilities accessible on consumer hardware while maintaining the model's ability to demonstrate reasoning, comprehension, and generation capabilities that surpass simple text storage.​

Parameter Compression in Float16 vs Float32 Format

Modern LLMs achieve significant storage efficiency through reduced precision formats, with Float16 (16-bit floating point) requiring half the memory of traditional Float32 (32-bit) representations while maintaining acceptable model performance. A 7 billion parameter model stored in Float32 would occupy approximately 28 GB (4 bytes per parameter), whereas the same model in Float16 format requires only 14 GB (2 bytes per parameter), effectively doubling the storage density without proportionally sacrificing accuracy. This precision reduction leverages the fact that neural network weights often don't require the full numerical range and precision of 32-bit floats, allowing the massive parameter counts of modern language models to fit within consumer-grade hardware memory constraints while preserving their knowledge encoding capabilities.

1-2 Bits Per Letter Information Density

Natural language exhibits remarkable redundancy, with theoretical information content ranging from approximately 1 to 2 bits per letter when accounting for predictability and context, far below the typical 8 bits used to store each ASCII character in raw text files. Modern language models exploit this inherent compressibility by learning the statistical patterns and dependencies between letters, words, and concepts, effectively encoding text at densities approaching these theoretical limits. This compression principle explains why a 4 GB quantized model can represent knowledge equivalent to tens of gigabytes of raw text: instead of storing each character with its full 8-bit representation, the model's parameters capture the underlying probability distributions that generate language, requiring only enough bits to distinguish between likely continuations rather than all possible character combinations. The float16 and 4-bit quantization techniques applied to LLM parameters align closely with this information-theoretic framework, allocating just enough precision to preserve the essential linguistic patterns while discarding the redundant bits that raw text storage necessarily includes.

source: excerpt of free Perplexity query using the "Page"-feature @ https://www.perplexity.ai/page/knowledge-storage-density-revo-KNmnAoUzQYKuNyFlZNPgUA

Saturday, November 15, 2025

Windows Recall: Security Flaws through RAG AI Agents

RAG (Retrieval Augmented Generation) systems integrated with Microsoft's Windows Recall feature present significant security vulnerabilities, as Recall's continuous screenshot capture and indexing capabilities could expose sensitive information through various attack vectors including prompt injection, vector index exploitation, memory poisoning, and proliferating AI agent ecosystems—all while alternative operating systems like Ubuntu offer more secure architectural approaches.

Gemini Workspace Vulnerability Parallels 

The Google Gemini for Workspace (also see: Gemini LLM) vulnerabilities that made headlines in the summer demonstrate precisely the kinds of risks that could affect Windows Recall when accessed by RAG-based AI systems. These real-world security issues provide a concrete example of how AI assistants can be compromised through indirect prompt injection when accessing various data sources.

Key vulnerabilities in Gemini for Workspace that mirror potential Windows Recall threats include:

• Indirect prompt injection attacks allow attackers to manipulate Gemini by embedding malicious instructions in emails, slides, and documents that the AI assistant processes, similar to how Windows Recall's screenshot database could be poisoned.

• Phishing exploitation where attackers can craft emails that, when processed by Gemini, display fake security alerts directing users to malicious websites, mirroring how Windows Recall's screenshot database could be used to trigger similar deceptive systems.

• RAG system manipulation in Google Drive, where Gemini behaves similarly to a Retrieval, Augment, and Generate system that can be compromised through injected content in shared documents, analogous to how Windows Recall's data could be accessed by compromised AI agents.

• Cross-platform vulnerabilities affecting various Google products (Gmail, Slides, Drive), resembling how Windows Recall data could be exploited across multiple AI-powered applications in the Windows ecosystem.

• Government-backed threat actors have already attempted to use Gemini for malicious purposes, including network reconnaissance and enabling deeper access in compromised networks, showing how sophisticated attackers could leverage AI assistants with access to sensitive data like that in Windows Recall.

Google acknowledges these security challenges and is developing countermeasures like "Project Mariner," which aims to help models prioritize user instructions over third-party prompt injection attempts, particularly those hidden in emails, documents, or websites. However, these mitigations remain works in progress.

​The Gemini vulnerabilities also highlight broader challenges in the "agentic era" of AI, including:

• Error propagation issues where compromised information in one part of a system can spread through interconnected AI agents

• Adversarial vulnerabilities that affect both individual AI assistants and multi-agent systems

• Unpredictability of emergent behaviors in complex AI systems with access to multiple data sources

Security researchers describe these AI assistant vulnerabilities as "raising significant concerns about the accuracy and reliability of its outputs", a warning equally applicable to any future integration between Windows Recall and AI agents. The real-world exploitation of Gemini for Workspace provides compelling evidence that the theoretical threats to Windows Recall discussed in previous sections are not merely speculative but represent genuine security risks that could soon materialize in the Windows 11 ecosystem.

Agentic Ecosystem Proliferation Risk

The Google Gemini for Workspace vulnerabilities that made headlines in the summer demonstrate precisely the kinds of risks that could affect Windows Recall when accessed by RAG-based AI systems. These real-world security issues provide a concrete example of how AI assistants can be compromised through indirect prompt injection when accessing various data sources. Microsoft's strategic push toward an "agentic future" for Windows 11 creates a perfect storm for security vulnerabilities by providing developers with powerful new tools to build AI agents that can access system data, including Windows Recall's extensive screenshot database. At Build 2025, Microsoft announced native support for Model Context Protocol (MCP), which offers "a standardized framework for AI agents to connect with native Windows apps, enabling apps to participate seamlessly in agentic interactions." This infrastructure, while innovative, dramatically expands the attack surface for RAG-based exploits. 

The integration of semantic search and knowledge retrieval APIs announced at Build 2025 specifically enables "developers to build natural language search and RAG (retrieval-augmented generation) scenarios in their apps with their custom data." When combined with Windows Recall's continuous screenshot capture functionality, this creates an unprecedented risk landscape where legitimate applications could inadvertently expose sensitive user data through poorly implemented RAG systems.

Microsoft's approach to security for these new capabilities appears primarily reactive rather than preventive. While they emphasize that "MCP server access will be governed by the principle of least privilege" and that "agents' access to MCP servers is turned off by default," these measures may prove insufficient against sophisticated attackers who can exploit the inherent vulnerabilities in RAG systems. The security model relies heavily on user control and transparency, which historically has proven inadequate when users don't fully understand the implications of granting permissions.

The introduction of "App Actions on Windows" further complicates the security landscape by allowing "app developers to build actions for specific features in their apps and increase discoverability." This feature, while enhancing functionality, creates additional entry points for potential exploitation. An attacker could craft a seemingly benign application that, once granted appropriate permissions, could leverage RAG techniques to extract sensitive information from Recall's database.

Windows 11's enhanced search capabilities that support "natural language queries across the OS" could be weaponized if compromised by malicious actors. Unlike traditional file system searches, these AI-powered searches create complex interaction patterns between applications and system data that are difficult to secure comprehensively. The semantic search functionality creates a database of file indexing that, while less invasive than Recall's complete screenshot approach, still presents significant security concerns when accessed by third-party RAG systems.

The "Agentic RAG" workflow demonstrated at Microsoft Reactor events shows how developers are being actively encouraged to combine "planning, tool use, and reflection to a RAG flow" using technologies like "OpenAI Function Calling with NL2SQL for structured data, Azure AI Search for unstructured data, and Bing Search API for live web search." This powerful combination of capabilities, while innovative for legitimate applications, also provides a blueprint for sophisticated attacks that could extract sensitive information from Windows Recall data.

The risk is particularly acute because Windows Recall creates a persistent, searchable history of user activity that remains on the device. Microsoft emphasizes that "Recall data is processed locally on your device, meaning it is not sent to the cloud and is not shared with Microsoft," but this local processing model doesn't protect against malicious applications that gain legitimate access to the data through approved APIs or exploit undiscovered vulnerabilities in the permission model.

For Windows 10 users who aren't upgrading to Copilot+ PCs with Windows 11, the risk profile remains significantly lower, as these advanced AI features require specialized hardware with neural processing units (NPUs) rated at 40 TOPS or higher. However, as the Windows ecosystem increasingly shifts toward these AI-enhanced experiences, the security gap between Windows 10 and Windows 11 users will widen substantially, creating a two-tier security landscape where Windows 11 users face significantly higher exposure to sophisticated RAG-based attacks.

Prompt Injection Through Screenshots

Prompt injection attacks through screenshots represent a particularly concerning threat vector for Windows Recall, as malicious actors could craft images containing invisible Unicode characters or strategically placed instructions that exploit LLMs processing the captured screenshots. When an AI agent or RAG system accesses Recall's database of screenshots, these embedded prompts can override the system's intended behavior without user awareness. Multimodal LLMs are especially vulnerable, as they can process both visible text and hidden instructions within images, potentially leading to data exfiltration through techniques like image URL manipulation that encode sensitive information in requests to attacker-controlled servers.​

The attack flow typically involves: (1) crafting malicious content with encoded instructions that remain invisible to users but are processed by LLMs, (2) ensuring this content appears in screenshots captured by Recall, and (3) triggering exploitation when an AI assistant accesses these screenshots during retrieval operations. This vulnerability is particularly dangerous because users may never notice the malicious content in their screenshots, creating a persistent attack surface that could lead to unauthorized access, data theft, or system compromise through what appears to be normal interaction with AI tools accessing Recall data.

DiskANN Vector Index Vulnerabilities

DiskANN, developed by Microsoft Research, offers high-recall vector search capabilities through its innovative Vamana algorithm, which builds a flat graph structure optimized for disk storage rather than requiring the entire index in memory. While this architecture enables efficient searches on massive vector datasets with reduced RAM requirements, it introduces potential security vulnerabilities when integrated with systems like Windows Recall. The algorithm's iterative post-filtering approach, which first retrieves items based on vector similarity before applying filters, could be exploited if malicious vectors are strategically inserted into the index, potentially circumventing security boundaries during the initial similarity search phase.

​From a security perspective, DiskANN implementations face several challenges:

• Its reliance on SSD storage for vector indexing creates a persistent attack surface where compromised vectors could remain undetected

• ​The expensive nature of incremental index updates may lead to delayed security patches when vulnerabilities are discovered

• While Azure implementations offer enterprise security features like Row Level Security and Transparent Data Encryption, these protections may be bypassed if an attacker can manipulate the vector similarity calculations that occur before security filters are applied

• The beam search mechanism that retrieves neighborhood data in batches could potentially be exploited to leak adjacent vector information through carefully crafted queries

​ 

RAG Memory Poisoning Attacks

RAG memory poisoning represents a sophisticated attack vector where malicious content is injected into knowledge databases that RAG systems rely on for accurate information. Research shows that just five carefully crafted documents in a database of millions can successfully manipulate AI responses 90% of the time. These attacks can be formally defined as targeted poisoning attacks where an attacker injects poisoned texts into a knowledge database to make the RAG system return predefined answers when queried with specific inputs.

[...]

Real-world examples include the Microsoft 365 Copilot exploit chain that combined prompt injection, automatic tool invocation, ASCII smuggling, and hyperlink rendering to access sensitive information, and AgentPoison, which targets LLM agents by poisoning their long-term memory or RAG knowledge base. Multimodal RAG systems are equally vulnerable, as attackers can manipulate text while keeping images fixed, effectively reducing multimodal attacks to text-based ones. In the context of Windows Recall, this vulnerability is particularly concerning as poisoned documents could persist in the knowledge base, creating long-term security risks when accessed by AI assistants.​

Ubuntu Security Advantage

Ubuntu 24.04 LTS has emerged as a significantly more secure alternative to Windows 11, particularly in light of the controversial Windows Recall feature. Security testing reveals that Ubuntu maintains a fundamentally different security architecture that doesn't rely on the same vulnerable approaches as Windows.

When subjected to identical vulnerability scans, Ubuntu 24.04 LTS demonstrated remarkable security resilience even with its firewall disabled. The system reported only seven informational notifications with no critical vulnerabilities, alerts, or failures. This contrasts sharply with Windows 11, which showed 40 informational notifications and a medium vulnerability when its firewall was disabled—including an SMB signature issue dating back to January 2012 that remains unpatched.

Ubuntu's security advantage stems from its architectural approach: rather than relying on firewall configurations to block access to listening ports (as Windows does), Ubuntu simply doesn't enable listening ports by default until corresponding applications are installed. This fundamental difference means that even if Ubuntu's firewall is compromised, the attack surface remains minimal, whereas Windows 11 becomes significantly more vulnerable without its firewall protection.

The Windows Recall feature introduces additional security concerns that Ubuntu users simply don't face. Microsoft's implementation of Recall, which captures screenshots every four seconds and uses OCR to index all viewed content, creates a persistent security risk that security experts have flagged as problematic. Despite Microsoft's claims that "processing and data storage is local" and that "the user has full control over all settings," the feature represents a significant security liability.

As one security expert noted: "Can you see how this would be a security nightmare? If a hacker were able to access your PC, they might be able to gather your banking records or other personal information." This concern is amplified by Microsoft's handling of the feature, with reports indicating that "Recall is incorrectly listed as an option under the 'Turn Windows features on or off' dialog in Control Panel," raising questions about users' ability to fully remove it.

The security community's response has been telling, with many technical users migrating to Linux distributions like Ubuntu. As one Windows developer who switched to Ubuntu 24.04 LTS noted, the transition was motivated by privacy concerns related to Windows 11's increasing AI integration and data collection practices. This migration trend is likely to accelerate as Windows Recall moves toward mainstream rollout, with security-conscious users seeking alternatives that don't create similar persistent attack surfaces.

For organizations implementing zero-trust security models, Ubuntu's minimal attack surface and transparent security architecture provide significant advantages over Windows 11's increasingly complex AI-integrated environment.[...]

[...]

As Windows 11 continues integrating AI features like Recall that create new attack vectors, Ubuntu's security advantage is likely to widen further, making it the preferred choice for security-focused users and organizations concerned about protecting sensitive data from increasingly sophisticated attack methods.

source: excerpt of free Perplexity query using the "Page"-feature @ https://www.perplexity.ai/page/windows-recall-security-flaws-Q5a7MAJWTn.KWGoDocbbmA

Tuesday, September 16, 2025

Local LLM-Inference on Gaming-GPU — Environmental & Privacy Benefits

Local Large Language Model deployment through platforms like LM Studio offers compelling environmental and privacy advantages over cloud-based inference, eliminating the substantial water footprint of data center cooling systems while ensuring complete data sovereignty for sensitive computational tasks. Gaming PC inference with pre-trained models achieves superior carbon efficiency by avoiding the power usage effectiveness overhead of commercial data centers and the computational burden of real-time web-scraping operations, while local deployment fundamentally transforms privacy protection by creating air-gapped environments where confidential information never leaves organizational boundaries.

Pre-trained Model Efficiency Advantages

Gaming PC deployment of pre-trained LLMs achieves superior carbon efficiency through several architectural optimizations:

• Static Model Loading: Pre-trained models loaded once into VRAM eliminate the continuous retrieval and caching overhead associated with dynamic web-scraping operations, reducing baseline energy consumption by avoiding constant network polling

• Elimination of Real-time Processing: Local pre-trained inference bypasses the computational overhead of web crawling, content parsing, and real-time data integration that characterizes browsing-enabled services

• Direct Parameter Access: Gaming GPUs access model parameters directly from local memory at 103103 times faster speeds than network-dependent cloud architectures, minimizing latency-induced energy waste

• Optimized Batch Processing: Local hardware enables aggressive batching strategies that maximize GPU utilization efficiency without the load balancing constraints of distributed cloud infrastructure
  
  

Scale-Dependent Environmental Mathematics

Recent benchmarking data reveals dramatic energy consumption disparities between deployment architectures, with implications that scale exponentially with usage patterns:

• High-Intensity Models: DeepSeek-R1 and o3 consume 33.634 Wh and 39.223 Wh respectively per long prompt on cloud infrastructure—over 70 times the consumption of efficient models like GPT-4.1 nano at 0.454 Wh

• Gaming PC Baseline: A single long query to resource-intensive cloud models consumes equivalent electricity to running a 65-inch LED television for 20-30 minutes, while local gaming GPU inference operates at fractional power levels

• Aggregate Impact: Scaling GPT-4o's 0.43 Wh short query consumption to 700 million daily queries results in electricity use comparable to 35,000 U.S. homes annually
  
  

Infrastructure Overhead Elimination

Local gaming PC deployment fundamentally eliminates the environmental multipliers that characterize cloud-based inference:

• Power Usage Effectiveness (PUE): Gaming PCs avoid the 1.2-2.0 PUE overhead of commercial data centers, where cooling and infrastructure consume additional energy beyond direct computation

• Water Usage Elimination: Local air-cooled systems eliminate the 1.8 liters per kWh water consumption of data center cooling systems, avoiding freshwater evaporation equivalent to annual drinking needs of millions

Regional Carbon Intensity: Gaming PC users in low-carbon electricity regions achieve substantially lower emissions than cloud services deployed in carbon-intensive data center locations

Computational Paradox at Scale

The research identifies a critical environmental paradox: while individual cloud queries appear efficient, their global scale drives disproportionate resource consumption through Jevons Paradox—as AI becomes cheaper and more accessible, total usage expands, intensifying environmental strain despite per-query efficiency improvements. Local gaming PC deployment with pre-trained models circumvents this scaling problem by maintaining consistent per-query emissions regardless of global usage patterns, as each user bears their own computational and environmental costs directly rather than contributing to aggregate cloud infrastructure demand.

The carbon intensity differential becomes particularly pronounced for users requiring multiple daily interactions, where local pre-trained model deployment offers both environmental sustainability and complete data sovereignty without the continuous environmental burden of maintaining globally distributed inference infrastructure.

Privacy Protection with Offline LLMs

Offline LLM deployment fundamentally transforms privacy protection by eliminating data transmission to external servers, creating an air-gapped computational environment where sensitive information never leaves the user's direct control. Data Protection Authorities recognize that traditional cloud-based LLM services introduce significant privacy risks throughout the model lifecycle—from training data collection to inference logging—where user queries, personal information, and proprietary content become permanently accessible to service providers and potentially exposed through data breaches or subpoenas. Local inference through platforms like LM Studio ensures that confidential documents, internal communications, and sensitive analytical tasks remain within organizational boundaries, addressing fundamental concerns about data sovereignty and compliance with regulations like GDPR where pseudonymized data still requires careful handling.

The privacy advantages extend beyond simple data retention to encompass protection against sophisticated inference attacks and model-based privacy violations. Research demonstrates that cloud-based LLMs can inadvertently memorize and reproduce training data, creating risks of sensitive information leakage through carefully crafted prompts. Local deployment eliminates these external attack vectors while providing complete audit trails and control over data processing workflows. Organizations processing regulated data—such as healthcare records, financial information, or legal documents—benefit from the ability to leverage advanced language capabilities without exposing protected information to third-party analysis or potential misuse.

source: free Perplexity query using the "Page"-feature @ https://www.perplexity.ai/page/local-llm-inference-on-gaming-TVbgCH8TR8KnA6y2ULGixQ

Wednesday, July 23, 2025

Potential SkyNet-Level Threat: Exascale AI Supercomputer NVIDIA GB200

NVIDIA's GB200 NVL72, a rack-scale supercomputer connecting 36 Grace CPUs and 72 Blackwell GPUs in a single NVLink domain, delivers unprecedented computational power with 1.44 exaFLOPS of AI compute and 30x faster real-time inference for trillion-parameter language models compared to previous generations, raising concerns about distributed AI networks potentially approaching artificial general intelligence thresholds that parallel fictional scenarios like SkyNet.

The GB200 represents a significant departure from the B200 in terms of scale and interconnect capabilities. While the B200 is designed as a versatile AI accelerator for diverse workloads, the GB200 specifically targets hyperscale AI training with its purpose-built architecture for trillion-parameter models. This specialization makes it particularly suited for the kind of massive, distributed intelligence systems depicted in science fiction.

A theoretical cross-country GB200 network would benefit from several technical advantages:

• Unified memory coherence across geographically distributed systems, allowing a single AI model to maintain awareness across multiple physical locations

• Low-latency communication between nodes, with minimal performance degradation even across long distances when using dedicated high-speed interconnects

• Fault tolerance through distributed processing, making the system resilient against localized failures or attacks
  
  

The power efficiency improvements in the Blackwell architecture — being twice as fast at training and five times faster at inference while using less energy — make large-scale deployments more feasible from both economic and infrastructure perspectives. This addresses one of the traditional limitations on AI system scale: power consumption and cooling requirements.

What makes this scenario particularly concerning from a SkyNet perspective is that such a network could potentially achieve computational capabilities approaching artificial general intelligence (AGI) thresholds. With each GB200 capable of processing trillion-parameter models in real-time, a nationwide network could theoretically support models with parameters in the quadrillions — far exceeding human neural complexity.

The technical risks associated with such deployments extend beyond the packaging and reliability concerns noted with previous generations. A distributed GB200 network would introduce unprecedented challenges in terms of control mechanisms, oversight, and the potential for emergent behaviors not anticipated by system designers. Unlike fictional depictions, the threat wouldn't necessarily come from malicious intent, but rather from the inherent complexity and potential for unintended consequences in systems operating at this scale and speed.

Industry analysts have already begun questioning whether NVIDIA's high-end GPU strategy is sustainable, with some comparing it to IBM's historical trajectory. The $30,000+ price points for these GPUs represent a "luxury solution" that may eventually give way to more specialized, purpose-built AI accelerators. However, this transition period — where extremely powerful general-purpose computing resources are being deployed at scale—represents a unique moment of both opportunity and risk.

NVL72 Exascale AI Capabilities

The GB200 NVL72 represents a quantum leap in computational capabilities, delivering 1.44 exaFLOPS of AI compute at FP4 precision. This exascale performance is achieved through a sophisticated architecture where 72 Blackwell GPUs function as a single, massive GPU with 130TB/s NVLink bandwidth. The system's unified memory approach creates a 30TB pool accessible across all GPUs, eliminating traditional communication bottlenecks that plague distributed systems. For AI workloads, this translates to practical benefits: a trillion-parameter model that processed 3.4 tokens per second on H100 GPUs can now handle approximately 150 tokens per second per Blackwell GPU — a 30-fold improvement that enables real-time inference for previously unwieldy models.

The technical specifications reveal the system's versatility across precision formats:

• FP4 Tensor Core: 1,440 PFLOPS (with sparsity support)

• FP8/FP6 Tensor Core: 720 PFLOPS

• FP16/BF16 Tensor Core: 360 PFLOPS

• TF32 Tensor Core: 180 PFLOPS

• FP64: 2,880 TFLOPS
  
  

This precision flexibility makes the NVL72 suitable for diverse applications beyond AI, including scientific simulations, real-time analytics, and computational research that previously required weeks to complete. In practical terms, a model like GPT-4 that required 90 days of training with 25,000 A100 GPUs could theoretically be trained in less than 2 days using 100,000 GB200 NVL72 systems.

72-GPU NVLink Domain Architecture

The GB200 NVL72's revolutionary performance stems from its sophisticated interconnect architecture that creates a unified 72-GPU compute domain. The system employs a flat, single-tier NVLink topology where each Blackwell B200 GPU connects to the NVSwitch fabric through 18 dedicated NVLink ports. The rack contains 18 compute trays (1U each) housing 36 Bianca boards (each with one Grace CPU and two B200 GPUs) and 9 NVSwitch trays with specialized switch chips. This arrangement enables any GPU to communicate with any other GPU in the rack through just a single switch hop, maintaining consistent low-latency communication across all 72 GPUs.

The NVSwitch fabric delivers an aggregate bandwidth of 130 TB/s across the system, creating what effectively functions as a single massive GPU rather than a distributed cluster. For larger deployments, NVIDIA offers the NVL576 configuration, which interconnects 8 NVL72 racks to scale to 576 B200 GPUs. While this larger configuration requires two NVSwitch hops between racks, the additional latency remains negligible for most training workloads and only minimally impacts inference scenarios requiring extremely high interactivity. This architecture represents a significant advantage over competing designs that rely on direct GPU-to-GPU connections without switches, which typically result in reduced accelerator-to-accelerator bandwidth.

30X Faster LLM Inference

The GB200 NVL72's claim of 30X faster real-time inference for trillion-parameter LLMs represents a paradigm shift in AI deployment capabilities. This dramatic performance improvement is enabled by several architectural innovations:

• Second-generation Transformer Engine with FP4 AI precision support, utilizing new microscaling formats that maintain accuracy while significantly boosting throughput

• Increased parameter bandwidth to HBM memory, allowing larger models to fit per GPU

• Advanced precision formats including community-defined microscaling and MX-FP6 that optimize both accuracy and throughput specifically for LLMs and MoE models

• Elimination of communication bottlenecks through the unified 72-GPU NVLink domain with 130 TB/s compute fabric
  
  

In practical terms, this translates to approximately 116 output tokens per second per GPU for the GPT-MoE-1.8T model, compared to just 3.5 tokens with the previous-generation HGX H100. This performance leap enables real-time inference with token-to-token latency (TTL) of 50 milliseconds and first token latency (FTL) of 5 seconds, even with massive context windows of 32,768 input tokens and 1,024 output tokens. Such capabilities fundamentally change what's possible with large language models, enabling truly interactive experiences with trillion-parameter AI systems that previously required significant compromises in response time or model complexity.

Fun Fact: Enterprise-D Computing Equivalence

• Space Efficiency: Each rack has a compact 10U form factor, with all 21 racks occupying only a fraction of a single deck within one Enterprise-D computer core.

• Cooling Compatibility: The 's liquid cooling requirements

• would be easily handled by the Enterprise-D's environmental systems.

• Technological Progress: 24th-century fictional computing power is achievable today with just 21 racks of commercially available hardware.

• Remaining Differences: Modern systems still lag in power generation, miniaturization, and the fictional isolinear optical processing technology compared to silicon-based computing.

This comparison highlights the remarkable pace of real-world computing advancement. What was once considered science fiction computing power from the 24th century is now achievable with just 21 racks of commercially available hardware in the early 21st century. The primary differences remain in areas like power generation, miniaturization, and the fictional isolinear optical processing technology versus our silicon-based computing.

source: free Perplexity query using the "Page"-feature @ https://www.perplexity.ai/page/nvidia-gb200-exascale-ai-super-eQbdxUvkTO2NHGMvzvYDrQ

Monday, June 9, 2025

Own witnessed incident: Bluetooth 5.2 Audio Routing Vulnerability

A critical Bluetooth 5.2 security vulnerability (possibly in Android's Combined Audio Device Routing feature?) allows unauthorized users to entirely automatically and involuntarily exploit Bluetooth headsets with built-in microphones that they are not paired with, enabling them to listen to phone-calls of strangers through these devices even when they're already paired with another device, as well as (which is even worse) causing a Denial of Service, making new Bluetooth 5.2 JBL earbuds entirely useless in these situations (other than us being allowed to spy on random peoples' phonecalls, but why the heck would one even want that?). All you have to do is pair the headset with your phone and avoid linking it to a Google account everytime your Android 15 phone tries to convince you with a popup-notification that linking it to an email-adress would be a safe thing to do...

I reported this incident to the Bluetooth SIG (via their official mail, security@bluetooth.com) on May 17th already but still haven't received any reply whatsoever yet... I sent them another mail regarding my previous mail just now. Will they ignore my report? 🤨 I'll update you as soon as I receive a reply!

So far I have not found any other reports of this issue other than with JBL Bluetooth 5.2 earbuds. So please test with some other brands and Android 15 with this procedure (not linking the Bluetooth-earbuds to a Google account), I'm very curious if this issue also already affects other manufacturers or not yet.

For now I sadly cannot recommend anyone anymore to chose Bluetooth 5.2 JBL earbuds because of this still unconfirmed Denial of Service issue.

view blog post archive 11.2024 - 06.2025

👆